In this session, we will learn about variables, groups, stimuli design and counterbalancing, all of them are crucial aspects of experimental design,

Types of variables

There are two types of variables that we need to consider:

Within- and Between-subject Variables

Creation of stimuli

You experiment is only as good as your stimuli are; this is why creating stimuli is usually the longest and most arduous part of designing an experiment.

Because we are taking a hypothesis-driven approach, we will design our stimuli based on the experimental manipulation we want to investigate.

Here is a real life example:

A substantial body of research has shown that, in ambiguous relative clauses containing the preposition with, disambiguation is preferred towards the second noun phrase (NP2). In the example below, this thematic preposition creates a strong preference to chunk the adjective and NP2 for disambiguation:



Now, in Spanish, we can introduce a second manipulation towards disambiguation using gender:



In the first sentence, the gender manipulation goes in line with the preference for NP2 attachment, while the second sentence violates it

First, what are the conditions? + Conditions + NP2 Masculine + NP2 Feminine + NP1 Masculine + NP1 Feminine


What confounding factors might we need to take into consideration when creating the stimuli? Confounds occur when the levels of the independent variable vary directly with some other factor that is not of interest to the hypothesis of the study but nevertheless has an impact on the dependent variable.

Answer Correct, some examples are:

  • Linguistic factors
    • Orthographic length
    • Lexical frequency
    • Cognate status
    • Language-specific aspects (e.g., transparent gender? cliente vs. alto/a)


Let’s see another real life example:


As you can see, the nouns (NP1 vs. NP2) in this list have columns with attributes such as the ones we mentioned above. The colors blue and green represent matching because these nouns were matched by list, what does this mean?

Answer To avoid confounds, we must identify factors that may be relevant and control for them. For example, we can choose a subset of the nouns in each group that are matched for lexical frequency. That is, our stimuli consist of pairs of nouns, one from each group, with similar lexical frequencies. We can also use statistical modeling to see if other factors have an effect over and above frequency.

IMPORTANT

Before moving on to other aspects of experimental design it is important to consider the following question: How many sentences per condition should I have?

The general rule of thumb is:

In addition, you want at least 30 participants, unless you’re doing a production study.

Counterbalancing

Counterbalancing is crucial in repeated measures/within-subjects designs because participants are seeing the same conditions multiple times (although the items are different).

While counterbalancing will not completely eliminate confounds like order effects, it does help distribute them evenly across all experimental conditions such that their influence is balanced and does not confound the main effects due to the independent variables.

Thus, counterbalancing is meant to control:


Counterbalancing can be obtained through different designs, the most common in Linguistics is within-subjects counterbalancing. This can be carried out through a Latin Square or complete counterbalancing:

Latin Square

A Latin square is a type of permutation. Permutation involves deriving all possible combination of a sequence. If we start with the sequence ABCD (x4) this would yield 4 possible permutations (for a total of 16 items, 4x4):

  • ABCD
  • BCDA
  • CDAB
  • DABC


Let’s see a real-life example:

Because we have 4 conditions in the current experiment, this yields a total of 80 stimuli (4x20).

We will create 4 lists of counterbalanced materials. Each subject will be assigned to an alternating list (e.g., subject 1=list 1, subject 2=list 2, and so on).

We have created a table were order and lists are columns.

Then we assign each batch of n=20 stimuli to an order in each list alternating the starting point (see List 1 = ABCD). In doing so, we obtain all possible permutations of items per condition, per list, such that a participant never sees the same item in more than one condition.

NOTE: implementing Latin Squares in experiments with an odd number of conditions is quite challenging.

Complete counterbalancing

This is another form or simple permutation, it is easier than Latin Squares and can be easily implemented in experiments with odd number of conditions.

In this case, we have created a table were conditions are columns and lists are rows.

Then we assign each batch of n=20 stimuli to a condition in each list alternating the starting point (see column A). In doing so, we also obtain all possible permutations of items per condition and per list, such that a participant never sees the same item in more than one condition.

Randomization

We use this method to deal with variation that is not inherent to our sample but, rather, a by-product of our experimental design (mostly). By eliminating this variation we ensure that the results we observe are indeed cause by the effect of our experimental manipulation.

Examples of the kind of variation randomization helps deal with:


There are two main types of randomization:

Resources